Studies of Confidence in Software Cost Estimation Research Based on the Criterions MMRE and PRED

Dan Port (UHawaii) Vu Nguyen (USc) Tim Menzies (WVU) [email protected], [email protected], [email protected] Jan 19, 2009 ABSTRACT Confidence in cost estimation is different than model accuracy. It is related to the significance of results based on model accuracy measures such as MMRE and PRED. A lack of confidence places uncertainty in the accuracy for predicted values and the conclusions based on accuracy measure. This work is a critical empirical and analytic investigation of standard error in the accuracy criterions MMRE and PRED. Center in this is the conclusion instability problem which we show can be resolved and greater confidence can be found in the use of model accuracy criteria in practice. Five published studies are replicated and extended and their results are recast with consideration of standard error. We explore a number of practical applications such as estimating data size needed for confidence, reliability of estimators, and outlier detection. All results are based on publicly available data sets and models from the PROMISE repository. As a secondary contribution, we investigate the efficacy of the popular model accuracy criterion PRED, which unlike MMRE has received relatively little investigation.